I will first review some methods for learning, then go to the technique that makes learning DNNs work. Along this line, I will focus on analysis from the perspective of optimization. The goal is to understand why and how a DNN functions. The outline roughly covers:
1. History and models of neural networks.
2. Why deep? Universality theorem for shallow/deep neural networks.
3. Optimization algorithms: back-propagation, block-coordinate descent
algorithms.
4. Classification problem: dimension reduction methods, support vector machines
and kernel methods. Un-supervised learning and auto-encoder.
5. Generative Adversarial Network.
6. ResNet, Mobile Net, U-nets, and RNN.
7. Using DNN to solve problems: the inverse problem, the forward inference
problem, and the generalization.